Data Mining - An Industrial Research Perspective

Data mining has burst into the limelight recently, thanks to a series of key application successes [2]. Big business and industry has raised the stakes by investing in this nascent technology, and has laid down high expectations for this emerging area. Although the methods and systems for data mining are based upon and in uenced by years of classical work in statistics, pattern recognition, information theory, and machine learning, a combination of factors has caused the recent resurgence of interest. These factors include the availability of high volumes of on-line enterprise data, inexpensive access to high performance computational resources, and continuing impressive advances to the underlying data analysis algorithms. This article describes activities that center around this emerging area of technology, with a focus on research in progress, and applications being pursued. 1 Current State of the Art Just what exactly is data mining? At a broad level, it is the process by which one extracts accurate and previously unknown information from large volumes of data. This information should be in a form that can be understood, acted upon, and used for improving decision processes of the data owning entity. Obviously, with this de nition, data mining is a technology that encompasses a broad set of technologies, including data warehouses, database management, data analysis algorithms, and visualization. The crux of the appeal for this new technology lies in the data analysis algorithms, since they provide the automated mechanisms for sifting through these large volumes of data for extracting useful information. The analysis capability of these algorithms, coupled with today's data warehousing and database management technology, make it possible to mine and extract useful knowledge from very large business and industrial data. The data analysis algorithms (or data mining algorithms, as they are more popularly known nowadays) can be divided into three major categories based upon the nature of their information extraction. These categories are as follows; predictive modeling (aka classi cation or supervised learning), clustering (aka segmentation or unsupervised learning), and frequent pattern extraction. The data representation model for all these algorithms is quite straightforward. Data is considered to be a collection of records, where each record is a collection of elds. Using this tabular data model, the data mining algorithms are designed to operate on the contents, under di ering assumptions, and delivering results in di ering formats. Predictive modeling is based upon techniques used for classi cation and regression modeling. One eld in the tabular data set is pre-identi ed as the response or class variable, and these algorithms produce a model for that variables as a function of the other elds in the data set, pre-identi ed as the features or explanatory variables. If the response variable is discrete valued, then classi cation modeling is employed. If the response variable is continuous valued, regression modeling is employed. The principal problem being addresses by this family of algorithms is to be able to produce a predictively accurate function approximation for the response variable, by using the data set as examples of the relations between instances of explanatory variables and the response variable, in the presence of noise. Once produced, the model can be used to predict the value of a response variable, given the speci cations for the explanatory variables. This modeling work has it's roots in classical statistics [4, 8], although many recent advances have come from other areas, including pattern recognition, information theory, and machine learning. The important